We have discovered four X-ray bursts, recorded with the Rossi X-ray Timing Explorer Proportional Counter Array between 2003 September and 2004 April, that we show to originate from the transient magnetar candidate XTE J1810-197. The burst morphologies consist of a short spike or multiple spikes lasting ~1 s each followed by extended tails of emission where the pulsed flux from XTE J1810-197 is significantly higher. The burst spikes are likely correlated with the pulse maxima, having a chance probability of a random phase distribution of 0.4%. The burst spectra are best fit to a blackbody with temperatures 4-8 keV, considerably harder than the persistent X-ray emission. During the X-ray tails following these bursts, the temperature rapidly cools as the flux declines, maintaining a constant emitting radius after the initial burst peak. During the brightest X-ray tail, we detect a narrow emission line at 12.6 keV with an equivalent width of 1.4 keV and a probability of chance occurrence less than 4 x 10^-6. The temporal and spectral characteristics of these bursts closely resemble the bursts seen from 1E 1048.1-5937 and a subset of the bursts detected from 1E 2259+586, thus establishing XTE J1810-197 as a magnetar candidate. The bursts detected from these three objects are sufficiently similar to one another, yet significantly different from those seen from soft gamma repeaters, that they likely represent a new class of bursts from magnetar candidates exclusive (thus far) to the anomalous X-ray pulsar-like sources.